Papers with coherence measures
Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for topic model evaluation use automated measures modeled on human evaluation tests that are dissimilar to applied usage. |
| Approach: | They propose to use a novel experimental framework to evaluate topic models and assess their coherence for specialized collections in an applied setting. |
| Outcome: | The proposed framework is reflective of human evaluations using open labeling, typical of applied research. |
How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate summary coherence are often evaluated using disparate datasets and metrics. |
| Approach: | They propose to use automatic evaluation to evaluate coherence of summaries by selecting high-scoring candidates. |
| Outcome: | The proposed methods show that they can perform better on an even playing field. |
Fiction Flows: A Replication and Reinterpretation of Narrative Sequentiality (2026.acl-long)
Copied to clipboard
| Challenge: | a new study shows that imagined narratives exhibit higher "flow" than recalled narratives, but this advantage is not reducible to standard coherence measures. |
| Approach: | They propose a language-model-based measure of sentence-level predictability to measure narrative flow . they find that imagined stories flow better than recalled ones . |
| Outcome: | The proposed measure of sentence-level predictability is based on language models . it shows that fiction exhibits a robust sequentiality advantage over reality-bound genres . |